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SYSTEM AND METHOD FOR AUGMENTATION OF 
ENDOSCOPIC SURGERY 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

This invention relates to the field of endoscopic surgery. More 
specifically the invention relates to obtaining accurate positional 
information about an anatomical structure within a patient's body 
and using this information to accurately position endoscopic cam- 
eras and surgical instruments within the patient's body. 

2. Description of the Prior Art 

Systems have been developed to augment a human surgeon's ability 
to perform surgery on a patient by providing the surgeon with 
intraoperative images of anatomical structures within the patient's 
body. Typically, these systems comprise a specialized form of 
camera or medical telescope. Further, a class of these systems, 
which includes endoscopic and laparoscopic instruments, has re- 
duced the invasive nature of many surgical procedures. 

This class of systems has two salient characteristics in common: 
First, the surgeon using the system cannot directly manipulate the 
patient's anatomy with his fingers, and second, the surgeon cannot 
directly observe what he is doing. Instead, the surgeon must rely 
on instruments that can be inserted through a trocar or through a 
working channel of an endoscope. Often, since his hands and at- 
tention arc fully occupied in performing the procedure, the surgeon 
must rely on an assistant to point the endoscopic camera while the 
surgery is pci formed. 

To ameliorate the awkwardness of this arrangement, robotic aug- 
mentation devices have been developed for endoscopic surgery. One 
such device is described in detail in a copending application entitled 
"System and Method for Augmentation of Suigciy" serial number 
07/714,816 filed June 13, 1991 which is herein incorporated bv icf- 
crencc. 

Robotic augmentation devices can potentially greatly assist sur- 
geons during an operation. Robotic devices do not fatigue. Poten- 
tially, they can position medical telescopes and surgical instruments 
very accurately and can perform precise repositioning and repetitive 



functions. However, in order for these advantages to be realized, a 
number of problems need to be solved. The surgeon still needs to 
determine what motions the robotic device is to make and requires 
a means to communicate with the computer controlling the robot 
In a few cases, such as orthopaedic machining of bone or pre- 
planned excision of a tissue volume determined from preoperative 
medical images (such as CT Or MRI scans), these motions may be 
preplanned. However, in other cases, the surgeon needs to directly 
observe the patient's anatomy and interactively specify the motions 
to be made relative to anatomical features and the medical tele- 
scopes. In these cases, means of accurately locating anatomical 
features and instruments relative to the medical telescopes and to 
each other and of using this information to control the robotic aug- 
mentation aids are necessary. 

A specialized robotic device for stepping a rcscctoscope through a 
preprogrammed sequence of cuts in thranurcthral prostatectomies 
has been developed. However, this system docs not address the 
problem of providing the surgeon with a convenient means of con- 
trolling the view available through an endoscopic device or of pro- 
viding the surgeon with means of interactively manipulating 
surgical instruments in response to intraoperative imaging and other 
sensory information. 

There has been one attempt to provide voice control of a flexible 
endoscope in which servomotors attached directly to the control 
knobs of a commercial flexible endoscope were activated in response 
to voice commands by the surgeon. Difficulties of this approach 
include: (a) the surgeon (or an assistant) must still determine which 
direction to deflect the endoscope tip to provide a desired view and, 
consequently, must keep track of the relationship between the 
endoscope tip and the anatomical structures being observed; (b) 
these corrections must be made continually, distracting the surgeon 
from more important matters; and (c) the use of voice commands 
for this purpose is subject U> aiovs, potentially distracting to the 
surgeon, and may make the use of voice for communication between 
the surgeon and operating room personnel more difficult. 

Several research efforts are directed to providing improved mech- 
anisms for flexible endoscopes. These devices do not, however, 
simplify the surgeon's problem of controlling the endoscopic camera 
to obtain a desired view, either by himself or by communicating 
with a skilled operator. 

3. Statement of problems with the prior art 
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Unfortunately, the medical telescopes which are used in minimally 
invasive surgery have limited fields of view. As a result, only a 
small part of the anatomical feature hidden inside the patient's 
body can be viewed at a one time. Furthermore, surgical telescopes 
typically provide only a single vantage point at any one time and it 
is difficult to provide the desired view. 

Normally, to compensate for this limited field of view, a surgical 
assistant operates the telescope, reorienting it to produce many 
views of the anatomical feature. While doing this, the assistant 
must continuously keep track of the relative orientation between the 
telescope and the patient's anatomy in order to be able to quickly 
and correctly aim the telescope at the surgeon's request. He or she 
must also correctly interpret the surgeon's desires, which are not 
always evident from the surgeon's verbal comments. 

This creates a number of problems. Surgical procedures of this na- 
ture now require an additional highly-skilled person to assist the 
surgeon in manipulating the medical telescope because the surgeon 
is using both of his hands performing other tasks. The communi- 
cation that is required between the surgeon and the assistant in- 
creases the potential for an error while performing the surgery. The 
surgeon (and assistant) have to develop and keep a mental image 
of the entire hidden anatomical feature because the telescope can 
not capture the full image of the feature. Many telescopes, whether 
flexible or rigid, piovidc an oblique view, i.e., the direction of view 
is not coincident with the main axis of the telescope. This further 
exacerbates the difficulties of correctly aiming the telescope to 
achieve a desired view and increases the likelihood that the surgeon 
or the assistant could misconstrue the image presented or lose the 
orientation of the telescope with respect to the anatomical feature. 
Human fatigue contributes to a degradation of positioning of the 
telescope and/or of the interpretation of the images that (he tele- 
scope transmits. 

Accordingly, there is a need for a way to obtain accurate and reli- 
able information about the position and appearance of anatomical 
features hidden within a body. There also is a need for an appara- 
tus to accurately position and orient surgical instruments and/or 
medical telescopes within a body and to provide accurate informa- 
tion about their position with respect to hidden anatomical features. 
Further, there is a need to provide a reliable and accurate interface 
between the surgeon and his surgical instruments so that he can 
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accurately position these instruments with respect to an anatomical 
feature within a body without removing his hands from his instru- 
ments. 



OBJECTIVES 

An objective of this invention is to provide an improved method to 
obtain and display accurate information about the position of an 
anatomical feature within a patient's body. 

Also an objective of this invention is to provide an improved 
method of positioning endoscopic cameras and other surgical in- 
struments within a patient's body. 

A further objective of this invention is to provide an interface for a 
surgeon to accurately position an endoscopic camera and/or other 
surgical instruments within a patient's body without removing his 
hands from the instrument. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a schematic view of a system used for computer 
augmentation of surgical procedures. 

Figure 2 is a detail of Figure 1 showing a distal fine motion rota- 
tional manipulator. 

Figure 3 shows an embodiment of the invention using a stereoscopic 
visualization system. 

Figure 4 shows an embodiment of the present invention comprising 
two robotic manipulators. 

Figure 5 shows positions in 2D and 3D Cartesian coordinate sys- 
tems. 

Figure 6 shows the pin-hole mathematical model of a camera. 

Figure 7 shows a method of computing a position in three dimen- 
sions using two nonparallel camera vantage points. 
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Figure 8 shows the use of passive visual targets to determine a po- 
sition of a surgical instrument. 

Figure 9 shows a method of computing a position in three dimen- 
sions using two parallel camera vantage points. 

Figure 10 shows a method of using oblique medical telescopes. 



SUMMARY OF THE INVENTION 

The present invention is a method and apparatus for determining 
positional information about an object and then using this infor- 
mation to position instruments in relation to the object. The in- 
vention has many applications but is particularly useful when the 
object is hidden from view or in a location that is difficult to access. 
One preferred embodiment, used in endoscopic surgery, determines 
positional information about a designated anatomical feature which 
is hidden within a patient's body! The information is used to posi- 
tion surgical instruments in the body with respect to the anatomical 
feature. 

The invention first positions an instrument, e.g. a surgical instru- 
ment inserted inside a patient's body, at a desired position relative 
to a designated object (anatomical feature). The instrument is ca- 
pable of transmitting an image of the object to a computer which 
then determines positional information about the object by using 
various types of image processing. The information is then related 
to a human (e.g., a surgeon) or to a computer controlling a robotic 
apparatus. The positional information is used to position or repo- 
sition the transmitting instrument and/or other instruments relative 
to the designated object. 

To fuilhcr facilitate use of (he invention, a number of different 
output modes for conveying information from the imaging instru- 
ments and computer to humans in the operating room arc provided. 

To further facilitate use of the invention, input devices arc incorpo- 
rated on the inserted instruments so that a human user can input 
requests to the system while concurrently manipulating the instru- 
ment. Other methods of inputting requests to the system, such as 
voice recognition systems, arc also incorporated so that communi- 
cations with the system does not interfere with instrument manipu- 
lation. 
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DETAILED DESCRIPTION OF THE INVENTION 



Referring to Figure I, (.here is shown a schematic view of a system 
for use in computer augmentation of laparoscopic or similar proce- 
dures. The system generally comprises a manipulator apparatus or 
robot 242, a computer 243, a drive motor interface 244, a 
monoscopic monitor 247 with a suitable image processor 245 and 
graphics adaptor 246, a stereoscopic monitor 272 with suitable 
stereo display system 271, and a terminal 248 for connecting addi- 
tional input devices to computer 243. 

A manipulator similar to the manipulator 242, used in this pre- 
ferred embodiment, is described in detail in the copending U. S. 
application serial number 07/714,816 filed on June 13, 1991. 

Referring to Figures 1 and 2, the manipulator 242 comprises a 
proximal rectilinear manipulator 6 and a remote center-of-molion 
distal manipulator 240. The proximal manipulator 6 comprises 
three mutually orthogonal sliding motion sections 1, 2, and 3, which 
provide motion in the X 3 Y, and Z directions. Sections 1, 2, and 3 
are equipped with computer-controlled motorized drives 4 con- 
nected to motion interface 244 and also have manual locking clamps 
5. The remote ccntcr-of-motion distal manipulator 240 comprises 
rotational sections 7, 250, 251, and 252 to provide 0 P 0„ 6 y , and 
distal 0 Z rotational motion, and a slide motor 253 adapted to axially 
slide instrument 254. These sections are equipped with computer- 
controlled motorized drives 249 interfaced to motor interface 244 
and have manual locking clamps 255. Each of the moving sections 
of manipulator 242 can be actuated cither manually or under com- 
puter control and can optionally be locked by a manual locking de- 
vice. All the motorized drives 4 and 249 arc controlled by computer 
243 through motor interface 244. 

Referring to Figure 2, there is shown a schematic view of the distal 
fine motion rotational manipulator 240 with an instrument 241 in- 
serted through an incision into a patient's body. In the embodiment 
shown, the distal manipulator 240 provides a five dcgrcc-of-frcedom 
(0 P , 6 Xi 0 y , 0 Z , and d) remote centcr-of-motion wrist, which is sup- 
ported by the aforementioned proximal positioning system with 
three orthogonal linear degrees of freedom (X, Y, and Z). The 
proximal linear degrees of freedom arc used to place the centcr-of- 
motion M of the remote center-of-motion wrist at the position of 
insertion into the patient's body P. Any alternative mechanical 



YO9-92-080 



-6- 



structure (such as a SCARA manipulator, manufactured and sold 
by IBM) with sufficient degrees of freedom could be substituted for 
this purpose. 

The four distal revolute degrees of freedom and the sliding degree 
of freedom of manipulator 240 give the surgeon a five degree-of- 
freedom spherical work volume centered at the insertion point M. 
These degrees of freedom may be selectively locked or moved inde- 
pendently (manually or under computer control) to assist the sur- 
geon in achieving a desired precise alignment Furthermore, small 
motions within the work volume can be achieved with only small 
motions of the individual axes. The point M (i.e., the point at 
which the surgical instrument enters the patient) remains unaffected 
by any motions of the distal manipulator 240. Thus the manipula- 
tor may be moved through its work volume without requiring that 
the patient position be moved or that the size of the entry wound 
be enlarged. 

One consequence of this design is that motion of the proximal ma- 
nipulator 6 is not needed unless the patient is moved. Conse- 
quently, in a preferred embodiment, the motion of proximal 
manipulator 6 is disabled by manual locking and/or disabling of 
drive motors whenever an instrument is inserted into the patient 
In this mode, the control computer 243 interprets commands re- 
questing motion of manipulator 242 as follows. When a motion is 
requested, the control computer 243 attempts io satisfy the request 
by moving only distal manipulator 240. If the motion can be ac- 
complished in more than one way, the computer selects the motion 
that minimizes the motion of the most proximal revolute motion 
section 7 (i.e., it minimizes motion of d p ). If the motion cannot be 
accomplished perfectly, the computer selects the motion of distal 
manipulator 240 that most closely approximates the desired motion. 
Modes are available to select minimization of positional error of the 
tip of instrument 241, orientation error, or weighted combinations 
thereof. If the error is greater than a prcspccificd threshold amount, 
the control computer notifies the surgeon using synthesized speech, 
an audible alarm, or other means, and makes no motion unless the 
surgeon explicitly instructs it to proceed, using voice recognition or 
other input modality. One alternative embodiment might seek al- 
ways to minimize the total motion of the distal manipulator 240, 
again forbidding motion of proximal manipulator 6 whenever a 
surgical instrument held by the distal manipulator is inserted into 
the patient's body. Yet another might permit small motions of the 
proximal manipulator, so long as the center-of-motion M stays 
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within a specified threshold distance (e.g., 3 mm) of the original 
value. 

If desired, a flexible tip may be added to the distal end of instru- 
ment 241 to provide additional degrees of freedom. In the case 
where a viewing instrument such as instrument 254 is used, an ad- 
ditional degree-of-freedom in adjusting the gaze direction may be 
provided by adding an adjustable-angle mirror or prism to the distal 
end of the instrument. 

Referring again to Figure 1, the instrument 254, in the embodiment 
shown, includes a video camera 259 and a light source 277 con- 
nected to the instrument via a fiberoptic cable 278. The video out- 
put of the camera 259 is fed into the graphics adaptor 246, where 
it may be optionally mixed with graphics output from computer 243 
and displayed on monitor 247. The video output from the camera 
is also optionally fed into the image processing system 245, which 
analyzes the image produced by the camera and provides informa- 
tion to computer 243 about the relative position of the surgeon's 
instruments, the camera, and the patient's anatomy. The video in- 
formation from the camera may be also optionally supplied to the 
stereo display system 271, which can assemble a stereoscopic view 
of the patient's anatomy from two or more images taken from dif- 
ferent vantage points and display the image on the stereoscopic 
monitor 272. 

In one preferred embodiment, the stereo display system is a 
StereoGraphics CrystalEyes (trademark of StcrcoGraphics, Inc.) 
system, where the two video signals are displayed on a stereoscopic 
monitor which alternatively displays the left and right eye image at 
a frequency of 120 Hz, updating the video information for each eye 
60 times per second. The surgeon wears stereoscopic liquid crystal 
(LC) goggles 273, which arc synchronized with the monitor and al- 
ternatively block light from entering left and right eye such that the 
left eye receives only the video signal from the left camera and the 
right eye receives only the information from the right camera. The 
frequency of alternation between left and right images is sufficiently 
high such that the surgeon perceives no flicker but rather a contin- 
uous stereoscopic image of the patient's anatomy. Other stereo 
display technologies arc available and may be used. 

In the embodiment shown, the surgeon is using a second surgical 
instrument 260 inside the patient's body, which has passive visual 
targets 276 placed on it. These targets 276 are markings on the in- 
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strument and are chosen so as to be easily locatable by the image 
processing system 245 in the images supplied by the camera 259. 

The set of input/output devices attached to input/output interface 
248 of computer 243 shown in Figure 1 may include a computer 
voice recognition and synthesis system 267, a joystick 268 mounted 
on the surgical instrument 260 and a sterilized touch screen 269 
mounted on monitor 247. In the preferred embodiment the joystick 
is a small device, functionally identical to a 2D or 3D mouse, but 
designed such that it can be mounted directly onto a surgical in- 
strument and such that at least two degrees of freedom of motion 
can be specified by applying pressure on a small joystick protruding 
from the device. One implementation of such a device uses strain 
gauges to translate an applied pressure or force into incremental 
displacement or velocity information. In another embodiment, a six 
degree-of-freedom input device, such as Spaceball (A Trademark 
owned by Spaceball Technologies, Inc.) could be used used to 
specify motion in any of the six degrees of freedom. Such a device 
could be mounted on a surgical instrument, on the manipulator 
structure, or at any other convenient point. One advantage of 
mounting an input device such as a small joystick on a surgical in- 
strument is that the surgeon can easily manipulate the joystick 
without removing his hands from the surgical instrument, thus per- 
mitting him to provide information to the computer (for example, 
of a desired direction of motion of a medical telescope) without in- 
terrupting his work. 

Thc speech recognition and synthesis system 267 includes means of 
inputting information to the system, such as a (possibly head 
mounted) microphone 275, and a means of conveying information 
to the surgeon, such as a speaker 274. The speech recognition sys- 
tem 267 is capable of understanding a vocabulary of instructions 
spoken by the surgeon and can relate the information about the 
commands it has received to the computer 243/ The surgeon may 
use any of these modalities, cither separately or in combination, to 
position graphic objects on the monitor 247, to select commands or 
operating modes from menus, and to command motions of the ma- 
nipulator 242. 

Referring to Figure 3, there is shown an alternative embodiment of 
the system for computer augmentation of laparoscopic or similar 
surgical procedures. In this embodiment, the surgical instrument 
254a is a stereoscopic medical camera, which incorporates two in- 
dependent lens systems or optical fibers and is capable of transmit- 
ting two simultaneous images from the patient's body. The two 
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lenses are separated by a small (known) distance and are thus able 
to provide a stereoscopic image. One embodiment of such a device 
would comprise two side-by-side fiberoptic bundles or lens systems 
and one fiberoptic light channel. The assembly would be sur- 
rounded by a suitable cylindrical casing. The video signals from the 
two cameras 259a and 259b are fed into the stereo display system 
271 and displayed to the surgeon on a stereoscopic display monitor 
272. Using interface hardware known in the art, both video signals 
are also optionally supplied to the image processing system 245 and 
the graphics adapter 246. 

Another embodiment of the system is shown in Figure 4, where the 
system comprises two manipulators 240a and 240b, carrying surgi- 
cal instruments 241a and 241b, respectively. In one embodiment, 
one of the surgical instruments is a medical telescope, whereas the 
other instrument is a surgical tool, such as medical forceps. Since 
both instruments are attached to robotic devices, both can be ac- 
tively positioned under computer control. On the other hand, as 
with the single manipulator arm in the case above, either or both 
robots can be controlled manually -by releasing, adjusting, and rc- 
locking joint axes one at a time. In an alternative embodiment, both 
surgical instruments 24 1 a and 24 lb comprise medical telescopes or 
other means of transmitting an image outside of a patient's body. 
In such an embodiment, one of the instruments (for example, 241a) 
may also comprise a surgical tool such as a miniaturized surgical 
forceps. In this case, information from images taken at two vantage 
points may be combined to provide precise 3D information to assist 
in placing the surgical instrument precisely on the desired portion 
of the patient's anatomy. 

Referring again to Figure 1, the image processing system 245 may 
be used to locate features on the patient's anatomy of interest to the 
surgeon. In this mode, the surgeon would designate a feature of 
interest by any of a number of means to be explained below. On 
the surgeon's command, supplied via any appropriate input device 
attached to the terminal 248, the computer 243 would instruct the 
image processing system 245 to acquire an image and precisely lo- 
cate the designated anatomical feature. In one embodiment, a ref- 
erence image of the designated feature would be acquired in 
response to the surgeon's command and stored. Image con elation 
techniques would be used to locate the feature during surgery. In 
an alternative embodiment, synthetic reference images could be 
generated from computer reconstructions of preoperative medical 
images and models. Once a feature has been located, the manipu- 
lator 242 can be moved to place the feature at any desired position 
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in the camera field of view. If desired, an additional image may be 
acquired, the feature re-located, and a further adjustment made to 
refine the desired placement of the camera. This process may be 
repeated a number of times to "zero in" on a feature to any desired 
accuracy. Each of the foregoing steps is explained below. 

As a matter of nomenclature, we will in the following text refer to 
positional information in a number of ways. Unless otherwise 
specified, the terms "position" and "location" will be used inter- 
changeably. We will be referring to two-dimensional (2D) and 
three-dimensional (3D) positions. When referring to an image ob- 
tained by a single monoscopic camera, an "image location" or "im- 
age position" should be understood as a 2D location within the 2D 
image. Referring to Figure 5a, such a location A (within a 2D im- 
age 800) is given as a pair of coordinates When the image is 
stereoscopic, "image location" or "image position" should be under- 
stood as a 3D location within the volume of the stereoscopic image. 
Referring to Figure 5b, such a location B is described by a triple of 
coordinates (xj/,z). We will also refer to positions of anatomical 
features. Such features are part of the patient's anatomy and all 
references to "feature location" or "feature position" should be un- 
derstood as 3D positional information about the feature in question. 

In order to use and manipulate images of the patient's anatomy, 
images must first be acquired. Referring to Figure 1, this is done 
by feeding the live video signal from camera 259 into the image 
processing system 245 comprising at least one video digitizer. A 
video digitizer is a device capable of converting an analog video 
signal into a digital signal, which can be stored in computer memory 
and arbitrarily modified by the computer. Conversely, a video 
digitizer can also convert a digitized (and possibly modified) video 
signal back into analog form for display on a standard monitor. 

If positional information is to be extracted from images obtained 
by a camcra/lcns system, a mathematical model of the camera and 
the lens must be available to relate image points (i.e., points on the 
camera's imaging plane) to the corresponding world points (i.e., 3D 
locations in the actual environment). To a good approximation, a 
perfect camcra/lcns system can be modeled as a pin-hole system, il- 
lustrated in Figure 6. The figure depicts a camera with a lens 600 
positioned a distance f in front of the image plane 601. The quan- 
tity / is referred to as the focal length of the lens. A point 
W = (xj;,z) tying in the plane 602 a distance d = - z in front of the 
lens is imaged onto the image plane 601 at the location C = {x\y% 
where xjd = x'ff and yjd — y'jf. 
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Given the image coordinates (x\j>') of a world point, the above re- 
lationships constitute two equations in three unknowns (jc, y, and 
z) and are thus not sufficient to recover the 3D coordinates of the 
corresponding world point, W. Referring to Figure 7, the informa- 
tion obtained from a single image 601a from a first vantage point 
600a defines a ray 605a in 3D space originating at the image point 
C passing through the lens center 600a, and extending to infinity. 
By definition, the actual world point W lies somewhere on this ray, 
but additional information is needed to determine its exact location. 
If a second image 601b, taken from a second vantage point 600b 
(whose position and orientation with respect to the first vantage 
point is known), is available, then the corresponding image point 
C b in the second image and the location of the second vantage point 
600b define a second ray 605b in space, such that the world point 
W lies on this ray a well. Using known mathematical techniques, 
the two rays can be resolved in the same coordinate system and 
their intersection can be computed, giving the 3D world coordinates 
(jcj/,z) of the point W. 

Most camera lenses introduce distortions which causes the corre- 
spondence of world and image points to depart from the above 
pin-hole model. The process of calibrating the camera/lens system 
can estimate the nature and amount of such distortions and the re- 
sulting mathematical model can be used to effectively "undistort" 
the image points. The pin-hole camera model can then be applied 
to the undistortcd image. A number of techniques for calibrating 
camera/lens systems are known. 

As part of the interaction with a two-dimensional image of the pa- 
tient's anatomy displayed to the surgeon on a conventional monitor, 
the surgeon may wish to designate (i.e., point to) a particular image 
location within the displayed image. The surgeon may point to a 
particular image location by using any of the following means: (a) 
by positioning a surgical instrument equipped with a distinct and 
clearly visible visual target so that the image of the visual target on 
the display coincides with the desired image location, (b) by ma- 
nipulating a graphical object on the screen using an input device 
mounted on a surgical instrument (such as joystick 268 in Figures 
1 and 3 or a similar device), or (c) by manipulating a graphical ob- 
ject on the screen using a conventional mouse. In method (a) the 
visual target may consist of a brightly colored spot or a known ge- 
ometric pattern of such spots at a known position on the instrument 
(e.g., pattern 276 in Figures 1 and 3). The use of a bright color, 
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distinct from any color naturally occurring inside the patient's body, 
greatly simplifies the problem of locating artificial visual targets and 
lessens the chances of erroneous location of such targets. Such spots 
on the surgical instrument can be located using known image proc- 
essing techniques, involving thresholding (to isolate the spots from 
the rest of the image) and computationally determining the centers 
of the so obtained thresholded regions. In methods (b) and (c) the 
position of the feature of interest is taken as the final position of the 
graphical object. 

Once the 2D coordinates of an image location have been specified 
to computer 243, the computer can confirm the location by marking 
the location with a graphical object superimposed on the image. In 
one embodiment of this method of confirming an image location to 
the surgeon, 2D cross-hair cursors or 2D box cursors can be used 
to show the location of interest in the image. The "image", in this 
context, can be either a TV camera image or a computer generated 
graphical rendition of the anatomical area of interest. 

We have so far described a variety of methods for the surgeon to 
specify a particular 2D location of interest in a monoscopic image. 
We next discuss methods, such as image processing, to determine 
positional information about three-dimensional anatomical features 
and/or surgical instruments in the patient's body. 

Referring to Figures 1 and 3, if a stereoscopic display (live or static) 
of the patient's anatomy is available during the surgical procedure, 
then a surgeon can designate the desired 3D anatomical feature of 
interest by manipulating a 3D stereoscopic graphical object (cursor) 
on the stereoscopic display 272 until the graphical object is coinci- 
dent with the desired anatomical feature. Any of the appropriate 
aforementioned input devices and modalities 248 (such as the sur- 
gical tool mounted joystick or trackball, voice, etc.) can be used to 
specify the desired motion of the graphical object within the 
stereoscopic volume of the image. 

If the actual physical size of a designated object is known, its dis- 
tance from the viewing instrument may be estimated from the size 
of its image, as seen by the viewing instrument. Since we know that 
the feature lies on a ray originating at the center of image of the 
feature and passing through the vantage point as shown in Figure 
7, the position of the feature relative to the viewing instrument may 
then be computed. Let the size of the feature in the image be /, let 
the corresponding actual size of the feature be s, and let / denote the 
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focal length of the camera. The distance z from the camera lens to 
the feature of interest can then be computed as z — (fx s)jL 

Referring to Figure 8, in one embodiment, where passive visual 
targets 701 on a surgical instrument 700 are used, the position of a 
3D feature (e.g., a surgical instrument 700) can be determined as 
follows: At least three non collinear circular spots 701 of known di- 
ameter s are marked on the surgical instrument 700 (Figure 8a). 
Since the surgical instrument may have an arbitrary- orientation 
with respect to the camera, these spots will in general appear on the 
image plane as ellipses 705 (Figure 8b). However, the length of the 
major axis of each ellipse / will be the same as the diameter of the 
circular image that would be seen if the corresponding circular spot 
were presented at that same distance from the lens in such a man- 
ner that the plane in which it lies is perpendicular to the view axis 
of the camera. Let the length of the major axis of -the observed el- 
lipse as it appears in the image be / (Figure 8b). Then the distance 
of the spot from the camera lens can be computed from 
z = if x s)ji Having performed this computation for at least three 
spots and knowing the position of the spot pattern with respect to 
the tip of the surgical instrument suffices to compute the 3D lo- 
cation of the tip of the surgical instrument with respect to the cam- 
era. Other techniques, known in the art, permit calculation of the 
position and orientation, relative to the camera, of a pattern of five 
dots from the 2D positions of their centroids in the image obtained. 
Other patterns of dots or other visual targets can be used as well. 
The 3D location of the tip of the instrument relative to the camera 
may then be readily computed from the known position of the tip 
relative to the visual target. 

Additionally, stereo image processing may be used to precisely lo- 
cate 3D anatomical features. In one embodiment, image processing 
can be used in conjunction with a stereoscopic camera to locate an 
anatomical feature. Referring to Figure 3, surgical instrument 254a 
is a stereoscopic medical camera, comprising of two independent 
lens systems or optical fibers and is capable of liansmilling two si- 
multaneous images from the patient's body. The lenses arc sepa- 
rated by a small (known) distance d, as shown in Figure 9. The 3D 
position of the anatomical feature relative to the camera tip can be 
computed from the pin-hole camera model (Figure 6). Specifically, 
if the image plane locations of the center of the feature of interest 
in the two images arc denoted by y] = (j^, r ; ) and f 2 = (jc 2 , y 2 ), as 
shown in Figure 9, then the distance z of the feature center from the 
came ra lens can be computed as z = (f x d)jc, where 
c = J(x 2 - xi) 2 + (y 2 — j>[) 2 and / denotes the focal length of the 
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camera. Image correlation techniques or other image processing 
techniques known to the art may be used to locate features in im- 
ages. 

Referring again to Figure 9, in another embodiment, using only a 
monocular camera, image processing techniques can be used to de- 
termine the position of an anatomical feature in three dimensions 
as follows: A first image 601a of the anatomical feature is acquired 
and a reference representation (such as a multi-resolution image 
pyramid representation known in the image processing art) is 
stored. The manipulator 242 is used to displace the camera lens tip 
600a laterally by a known amount d, and a second image 601b is 
acquired. The center of the feature of interest W is located in the 
second image, using the reference representation of the feature, by 
means of correlation techniques (such as multi-resolution normal- 
ized correlation methods known in the art) and the 3D displacement 
of the anatomical feature from the camera tip may be computed as 
in the case above. Specifically, if the image plane locations of the 
feature of interest W in the two images 601a and 601b are denoted 
by fx — {xuy\) and fx — (x 2) j ; 2), respectively, then the distance z of 
the feature from the camera lens can be computed as z = (fx d)(c, 
where c = <J(x 2 — X\) 2 + (y 2 — j'i) 2 and / denotes the focal length of 
the camera. 

In another embodiment, the physical constraint of maintaining 
minimal translational motion of the telescope with respect to the 
port of entry into the patient's body may preclude laterally dis- 
placing the telescope to obtain a second image, as described above. 
Referring to Figure 7, in this embodiment, a first image 601a is ob- 
tained from the first vantage point and the center W of the feature 
of interest is located in the image at image location C a The telescope 
is then rotated by a small (known) amount about the port of entry, 
such that the desired feature is still within the field of view of the 
telescope, and a second image 601b is obtained. Note that the sec- 
ond vantage point has a different position and orientation than the 
first vantage point. The feature center W is located in the second 
image at image location Q. The 3D position of the feature center 
W is then obtained by computing the intersection of the rays 605a 
and 605b, as described previously. As above, image correlation 
techniques or other image processing techniques known to the art 
may be used to locate features in images. Alternatively, the surgeon 
may be asked to manually designate the image location of the fea- 
ture center in the two images using any of the means of designating 
image locations described previously. 
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Once a 3D feature has been designated and its 3D location suc- 
cessfully computed, computer 243 can confirm its location by 
marking the location with a 3D stereoscopic graphical object 
superimposed on the stereoscopic image of the area of interest. In 
one embodiment of this method of confirming 3D feature location 
to the surgeon, 3D cross-hair cursors or 3D box cursors can be used 
to show the feature's 3D location within the stereoscopic view vol- 
ume. The "image", in this context, can be either a TV camera im- 
age or a computer generated graphical rendition of the anatomical 
area of interest. 

Once the 3D positions of anatomical features are stored in com- 
puter 243, this information may be used to control the position and 
orientation of the camera tip relative to the features so as to provide 
any desired field of view. 

Referring to Figure 1, in one mode the surgeon can designate a first 
and a second 2D location in the image, using any of the means for 
designating 2D locations discussed above. The surgeon can then 
instruct the manipulator 242 (using any appropriate input device 
or modality as described earlier) to reposition the camera tip 266 so 
that the anatomical feature f h whose image appeared at the first 2D 
image location prior to camera motion, appears at the second 2D 
location in the image after the camera motion. The distance of the 
camera tip 266 from the anatomical feature f x remains constant 
during the camera motion. A special case of this mode is the case 
where the second 2D location is the center of the image. In this case 
the camera is repositioned so that the anatomical feature f { appears 
to move to the center of the displayed image, i.e., the camera is 
"centered" over the anatomical feature. 

In another mode the surgeon can specify a sequence of 2D locations 
in an image and instruct the manipulator 242 to move the camera 
tip 266, at a constant elevation, so that the camera traverses the 
path defined by the sequence of 2D locations in the image. In one 
embodiment, this sequence of image locations can correspond to 
image locations of distinct small anatomical features within the 
camera's field of view. In another embodiment, the sequence of 
image locations can correspond to image locations of a boundary 
of a large anatomical feature, such as a blood vessel. This mode of 
repositioning the camera can be viewed also as specifying the de- 
sired apparent motion of an anatomical feature (corresponding to 
the last 2D location in the specified sequence) with respect to the 
image. The term "apparent motion of an anatomical feature" is 



YO9-92-080 



-16- 



used to emphasize that the anatomical feature does not physically 
move, but only appears to move relative to the image due to the 
motion of the camera. Specifically, the execution of this mode pro- 
ceeds as follows: The sequence of 2D image locations is processed 
by computer 243 into a continuous path by the process of interpo- 
lation. The camera is then centered over the anatomical feature 
corresponding to the first designated 2D image location as described 
in the previous paragraph. The camera is then repeatedly posi- 
tioned so as to center each of the successive interpolated 2D lo- 
cations within its field of view, thereby effectively traversing the 
path as defined by the surgeon. The surgeon directly controls both 
the direction and speed of the camera motion by means of the sur- 
gical tool mounted joystick or any other appropriate input means. 

In another mode the surgeon can specify an increment of motion 
along the camera's axis of view and reposition the camera along this 
axis by the designated amount. The "axis of view" in this context 
is defined as the line joining the camera lens center and the point p 
on the patient's anatomy which appears in the center of the camera 
image. This mode effectively implements a zoom function with re- 
spect to a 3D anatomical feature, where the zoom factor (i.e., de- 
sired enlargement or contraction of the image of the anatomical 
feature) is specified by the surgeon interactively. In particular, this 
mode can be implemented by allowing the surgeon to interactively 
manipulate a graphical cursor on the screen whereby he can specify 
the desired zoom factor by enlarging or contracting one such cursor 
with respect to a reference cursor whose size docs not change during 
the zoom factor specification. Any appropriate input device 248 
can be used to manipulate the cursor object. Computer 243 then 
uses the relative geometry of the two cursors to compute the direc- 
tion and magnitude of the camera motion increment, which is nec- 
essary to effect the specified zoom factor. Alternatively, voice input 
can be used lo specify the zoom factor. Once the camera motion 
increment has been computed, computer 243 instructs manipulator 
242 to (slowly) reposition the camera tip 266 by (hat amount along 
the axis or view, thereby obtaining the desired zoom factor. Note 
that the point/;, as defined above, icmains at the center of the im- 
age throughout the zooming process. 

In another mode, the surgeon can directly control a desired direc- 
tion of motion of the camera vantage point via an instrument- 
mounted input device. In the preferred embodiment, this input 
device is a six dcgrcc-of-frccdom joystick. Using such a joystick, the 
surgeon can then arbitrarily reposition and reorient the camera in 
all six degrees of freedom simultaneously. By selecting different 
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subsets of the full six degree-of-freedom motion, a number of useful 
control modes can be implemented. In particular, if the 
translational controls of the six degree-of-freedom joystick are disa- 
bled or only a three degree-of-freedom input device is available, a 
camera motion control mode can be implemented, where the camera 
tip is constrained to move along the surface of an imaginary sphere, 
centered at the current anatomical feature of interest and having 
radius equal to the current distance of the camera tip from the fea- 
ture. In another embodiment, where only a two degree-of-freedom 
input device is available, any two of the six degrees of freedom can 
be controlled by the device at any given time. For instance, pressing 
a two degree-of-freedom joystick in the direction toward the tip of 
the instrument on which the joystick is mounted can be interpreted 
to mean "zoom in", and pressing away from the tip can mean 
"zoom out". Releasing the joystick can mean "stop". Similarly, 
exerting pressure or force on a two degree-of-freedom joystick in a 
direction perpendicular to the long axis of the camera can be inter- 
preted by computer 243 to mean a desired lateral motion of the 
camera at the current elevation in the direction of the exerted pres- 
sure. Additionally, the velocity of the camera motion can be made 
proportional to the amount of exerted pressure on the joystick. 

In another mode the surgeon can manipulate a graphical object 
superimposed on the image of the patient's anatomy to specify a 
desired view of a particular feature of interest. The camera is then 
automatically positioned to achieve the desired view. A particular 
implementation of this mode would proceed as follows: An image 
of the patient's anatomy is obtained and displayed to the surgeon 
on a display monitor. The surgeon is then allowed to designate a 
feature of interest in a 2D or 3D image, unless the desired feature 
has already been designated and is visible. Next the surgeon can 
interactively manipulate a graphical object (e.g., cursor, slider, etc.) 
superimposed on the image of the patient's anatomy on the display 
screen to specify the desired view of the feature of interest. For 
example, the view specification could specify the desired vantage 
point of the camera anywhere on the surface of a sphere of a given 
radius centered at the feature of interest. Computer 243 then com- 
putes the appropriate displacement of the camera and instructs the 
manipulator 242 to execute the motion, thereby obtaining the de- 
sired view of the feature of inlcrest 

If the surgical augmentation system comprises two independently 
controlled robotic systems, as illustrated in Figure 4, another mode 
of using the 3D positional information about the patient's anatom- 
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ical features to reposition a surgical instrument can be used, where 
the instrument being repositioned is the second surgical instrument, 
rather than the surgical telescope. In one embodiment of this in- 
vention, the second surgical instrument could be surgical forceps, 
which are repositioned such that the jaws of the instrument are co- 
incident with the current 3D anatomical feature and a tissue sample 
of this feature can thus be obtained by closing the instrument's 
jaws. 

Referring to Figure 10, the capability of interactively designating 
the desired view of a particular 3D feature of interest and letting the 
computer compute the resulting new location of the medical tele- 
scope is especially important in situations where the telescope's op- 
tics provide a lateral, rather than a straight-ahead (a = 0°) view. 
Telescopes with the direction-of-view anywhere between 30° and 
135° (with respect to the instrument's long axis) arc commonly used 
in laparoscopic and similar procedures. Figure 10 illustrates a tele- 
scope with the direction-of-view of a = 45°. Manually positioning 
such a telescope to achieve a desired view can be extremely difficult 
even for an experienced camera operator as the relative transfor- 
mations between the telescope, the patient's anatomy and the image 
coordinates become rather complex and unintuitive. However, 
adding a single rigid body transformation to the computational 
chain in the computer software accounts for the fact that the 
direction-of-view is different from 0°. In a particular implementa- 
tion, a coordinate frame F c is associated with a 0° telescope, and the 
computer keeps track of the rigid body transformations between the 
manipulator, the camera, and the various anatomical features of 
interest. The mathematical methods and techniques of representing 
and manipulating rigid body transformations are well known to the 
art of robotics and computer graphics. Camera motions needed to 
effect a particular zoom factor, for example, arc then computed 
relative to this camera frame F c . For the case of a non-straight 
telescope, such as the telescope in Figure 10, a new coordinate 
frame F c - is defined by rotating the frame F c through an angle of 
- a about a line passing through the center of the lens tip and par- 
allel to the A'-axis of the coordinate frame F r . The rigid body 
transformation C T C < relating the new camera frame F c - to the default, 
0° location of the camera frame F c , is used to account for the non- 
zero direction of view of the telescope. Using the transform F r - in 
place of F c in the computation of (he new desired position of the 
telescope for a particular desired view now rcsulLs in correct reposi- 
tioning of the telescope regardless of its direction-of-view. 
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The visual information transmitted from the patient's body and 
optionally augmented by image processing and computer graphics 
can be displayed to a surgeon in a number of ways. 

Referring to Figure 1, in one mode of information display, the im- 
ages of the patient's anatomy can be displayed to the surgeon as a 
combination of live and still images (a live image is an image ob- 
tained from the camera that is continuously updated with new in- 
formation, whereas a still image is not). In one embodiment of this 
mode, the image to be displayed on the monoscopic monitor 247 is 
produced as follows: A wide-angle monoscopic image of the pa- 
tient's anatomy is obtained using the surgical instrument 254 and 
displayed on the monitor 247 as a static image. The camera is then 
zoomed in for a closer view of the current feature of interest and a 
portion of this live TV image is displayed superimposed on top of 
the static wide-angle image. The static monoscopic view of the 
overall area of interest thus provides contextual information about 
the patient's anatomy under observation, whereas the live subimage 
shows a magnified detail area surrounding the current anatomical 
feature of interest. 

In an alternative embodiment of this display mode, the static wide- 
angle contextual information can be a computer-graphic rendering 
of the patient's anatomy. This graphical information can be derived 
from computer models of the patient's anatomy constructed on the 
basis of the information gathered during preoperative imaging and 
scanning. As before, a portion of the image surrounding the current 
anatomical feature of interest is replaced with a live magnified TV 
image of this area. Here, the computer generated image and actual 
live TV image arc merged into a single display image and must thus 
be properly registered with respect to each other to ensure proper 
correspondences of anatomical points and features between the two 
images. A number of techniques for achieving registration between 
images arc known to (he art. In the simplest embodiment, the 3D 
locations of a number of known anatomical landmarks represented 
in the computer model would be identified by 3D image processing 
techniques. The 3D locations of these landmarks can then be used 
to compute the appropriate perspective view for displaying the 
graphical modcL 

In another embodiment of this display mode, the static wide-angle 
contextual information can be a computer-graphic rendering of the 
patient's anatomy, as above. Similarly, a portion of the image sur- 
rounding the current anatomical feature of interest is replaced with 
a live magnified TV image of this area, as above. In addition, the 
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live TV image of the area of detail can be augmented by superim- 
posing static edge information, which can be derived either from a 
computer graphics model or as a result of image processing (edge 
extraction) on the TV image. The advantage of this display mode 
is that the superimposed edges highlight the ongoing changes within 
the area of detail reflected in the live TV image with respect to the 
previous (static) appearance of this area. 

In another embodiment of this mode of displaying information to 
the surgeon, the static wide-angle view of the overall area of interest 
can be displayed as a static stereoscopic image. Referring to Figure 
1, this is achieved as follows: A static image of the overall area of 
interest is obtained from a first vantage point using the surgical in- 
strument 254 and camera 259. The tip of the camera lens 266 is 
then displaced by a small known amount and a second static image 
of the area of interest is taken from this displaced vantage point 
The two images are then fed as input to the stereo display system 
271 and displayed on the stcroscopic monitor 272 as a static 
stereoscopic wide-angle view of the overall anatomical area of in- 
terest. In some cases where only the distal manipulator 240 is 
moved to displace the camera, there may be some small angular 
misalignment of the two images so obtained. Experiment has shown 
that this misalignment can often be ignored, since the human visual 
system is very adept at fusing slightly misaligned images. Alterna- 
tively, the misalignment can be largely compensated for by using 
image transformation techniques known in the art. Next, the cam- 
era is zoomed in for a close-up view of the current anatomical fea- 
ture of interest and a portion of the static wide-angle image is 
replaced by the magnified live monoscopic view of the anatomical 
feature of interest, as before. This results in an image, where the 
overall contextual information is a static stereoscopic image, pro- 
viding the surgeon with a sense of the global three-dimensional re- 
lationships within the viewing volume, and the area surrounding the 
current anatomical feature of interest, where the surgeon's concen- 
tration is focused, is magnified and displayed as a live monoscopic 
image. 

In a modification of the above mode of display, the live TV image 
of the area of detail can be augmented by superimposing static edge 
information, which can be derived cither from a computer graphics 
model or as a result of image processing (edge extraction) on the 
TV image. As described previously, the advantage of this display 
mode is that the superimposed edges highlight the ongoing changes 
within the area of detail reflected in the live TV image with respect 
to the previous (static) appearance of this area. 
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Referring to Figure 3, another embodiment of the present invention 
regarding display of visual information to the surgeon, uses the 
stereoscopic medical camera 254a to obtain a static stereoscopic 
wide-angle image of the overall anatomical area of interest. Then, 
as above, the stereoscopic camera is zoomed in closer over the cur- 
rent 3D anatomical feature of interest and a portion of the static 
image surrounding the feature of interest is replaced by a magnified 
live stereoscopic TV image as transmitted from the patient's body 
by cameras 259a and 259b. 

In order to emphasize the changes occurring within the area of de- 
tail, edge information corresponding to a previous state of the area 
of detail can be superimposed on the live stereoscopic image, as be- 
fore. 

Again referring to Figure 3, another embodiment of the present in- 
vention uses the stereoscopic medical camera 254a in conjunction 
with stereoscopic computer graphics to provide a display of the pa- 
tient's anatomy. In this embodiment, the static stereoscopic view 
of the overall anatomical area of interest is derived from computer 
models of the patient's anatomy and displayed on the monitor 272 
as a 3D stereoscopic graphical image via the stereo display system 
271. As above, the stereoscopic camera is then zoomed in closer 
over the current 3D anatomical feature of interest and a portion of 
the static graphical image surrounding the feature of interest is re- 
placed by a magnified live stereoscopic TV image as transmitted 
from the patient's body by cameras 259a and 259b. 

Again, in order to emphasize the changes occurring within the area 
of detail, edge information corresponding to a previous state of the 
area of detail can be superimposed on the live stereoscopic image, 
as before. 

Referring to Figure I, another mode of display of anatomical in- 
formation to a surgeon uses the monoscopic camera 254 to provide 
the surgeon with a live stereoscopic image of the patient's anatomy. 
In this mode, the information supplied to one of the surgeon's eyes 
is derived from computer models of the patient's anatomy and is 
displayed as a graphical image computed from the vantage point 
displaced a small known distance laterally from the current vantage 
point of the surgical instrument 254. The information supplied to 
the other eye is the live image of the patient's anatomy as provided 
by the camera 259 attached to the surgical instrument 254. In this 
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mode, one eye therefore receives static computer generated view of 
the patient's body, whereas the other eye receives a live image 
transmitted by the camera from a slightly displaced vantage point. 
If the computer-graphic model is properly registered with the actual 
anatomy, the human brain will fuse the two images into a proper 
3D stereoscopic image. 

In another embodiment of the above mode of display of anatomical 
information to the surgeon, image processing is used in conjunction 
with live video information to produce a live stereoscopic display to 
the surgeon. Referring to Figure 1, in this embodiment of the 
present invention, a first image of the patient's anatomy under ob- 
servation is obtained and transferred to the image processing system 
245. The camera tip 266 is then displaced laterally a small known 
amount and a second image is obtained from this second vantage 
point and transferred to the image processing system 245. The im- 
age processing system and known image processing techniques are 
then used to extract edge information from the two images. A 
stereoscopic display is then produced by supplying the stereo dis- 
play system 271 with only edge information in one of the input 
channels (left/right eye) and a live video signal with overlaid edge 
information in the other input channel (right/left eye). Subse- 
quently, only information to one of the two eyes is updated with live 
video as transmitted by camera 259. This provides enough infor- 
mation for the human brain to 'Till in" the missing information and 
interpret the image as a proper stereoscopic 3D image. 

Alternatively, a display mode as above can be used, where the edge 
information is not obtained by image processing, but rather derived 
from a computer graphical model of the patient's anatomy. 

Aside from visual information, the surgeon can receive non-visual 
information about the locations of features or the general state of 
the system as well. One non-visual channel of communication be- 
tween the surgeon and the system is the voice recognition and 
speech synthesis subsystem (267, Figure 1). For example, synthe- 
sized voice messages can be issued by the system to inform the sur- 
geon of the exact location of his surgical instrument with respect to 
an anatomical feature of interest. Likewise, synthesized messages 
confirming successful receipt of a voice command can be used to 
assure the surgeon that the system correctly interpreted his 
command(s). General system state or change of system state infor- 
mation can be relayed to the surgeon using synthesized voice as 
well. An example of this would be a synthesized speech message to 
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the surgeon stating the exact distance by which the camera was 
moved during a zooming operation. 

An alternative method of relaying non-visual information to the 
surgeon is tactile feedback. In one embodiment of this invention, 
tactile feedback conveyed to the surgeon through a hand-held or 
instrument-mounted input device (such as a joystick) can be used 
to alert the surgeon that he has positioned a graphical object or a 
surgical instrument in the vicinity of the current anatomical feature 
of interest. The tactile feedback can be delivered to the surgeon's 
hand or finger (whichever is in contact with the joystick) by instru- 
menting the joystick control with a computer controlled vibrator. 
When the vibrator is activated by the computer, the joystick control 
starts vibrating with appropriate frequency and amplitude, such 
that the oscillations are readily discernible by the surgeon, but do 
not distract him from his positioning task or otherwise interfere 
with his work. 
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CLAIMS 



We claim: 

■k A method determining positional information about an ana torn - 
ical feature within a patient's body comprising the steps of: / 



inserting a first surgical instrument into the patient's bodyythe in- 
strument having a means for transmitting an image out of the pa- 
tient's botiy; / 

designating an anatomical feature of interest; / 

transmitting anVnage of the designated anatomical feature out of 
the patient's bod\ / 

determining positional information about the/designated anatomical 
feature of interest by uVng image processing. 

2. A method of determining positional /information about an ana- 
tomical feature, as in Claim\, where mc anatomical feature is des- 
ignated by pointing with a )tecond/surgical instrument having a 
visual targeL \ / 

3. A method of determining positional information about an ana- 
tomical feature, as in Claim l/whercslhc anatomical feature is des- 
ignated by manipulating a/computc\ generated graphics object 
displayed on a video screer/supcrimposck on the image transmitted 
by the first surgical instrinticnt. \ 

4. A method of determining positional information about an ana- 
tomical feature, as/in Claim 3, where the computer generated 
graphic object is Manipulated by the use of a joystick mounted on 
a surgical instrument. \ 

5. A method yof determining positional information aaout an ana- 
tomical feature, as in Claim 3, where the computcr\pcncratcd 
graphic obfect is manipulated by the use of a force sensing device 
mounted/on a surgical instrument, \ 

6. A/friethod of determining positional information about an afca- 
tmmcal feature, as in Claim 1, where the information is provided!*) 




surgeon 
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A method uf Ueleiiuiii ing positional hrfunnation about an ana- 
ttomical feature, as in Claim 6, where the information is provided tc 
th\ surgeon in the form of synthesized speech. 

8. AViethod of determining positional information about an/ ana- 
tomical feature, as in Claim 6, where the information is provided to 
the surg\on in the form of tactile feedback. 

9. A methdd of determining positional information ah6ut an ana- 
tomical feature, as in Claim 6, where the infoi mation is provided to 
the surgeon irk the form of a computer generated Graphics object 
superimposed ojj an image obtained from the firs/ surgical instru- 
ment. 
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10. A method of determining positional information about an ana- 
tomical feature, as i\ Claim 8, where the tactile feedback is pro- 
vided in the form of vibrations on a surgical instrument held by the 
surgeon. 

11. A method of determining positional information about an ana- 
tomical feature, as in Claim V wherjz the computer graphics objects 
are displayed in two dimensions. 

12. A method of determining national information about an ana- 
tomical feature, as in Claim 9/whc^c the computer graphics objects 
arc displayed in three dimensions. 

13. A method of determining positiona\informa(ion about an ana- 
tomical feature comprising: 



inserting a first and/a second surgical instrument into the patient's 
body, each instrument having a means for transmitting an image 
out of (he palicn/s body; 

obtaining a fi/st image of the feature from a first\antagc point us- 
ing the first/urgical instrument; 

obtaining' a second image of the feature using (he sckond surgical 
instrument from a second vantage point, the second vantage point 
being/at a known position and orientation with respect to the first 
vantage point; 



Seating the anatomical fcatu re in both images; 
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computing the position of tho anatomical feature relative to the? 
vaniagc points using positional information about the feature in 
eachVnage, together with the known position and orientation ojAhe 
two vantage points with respect to each other. / 

14. A method of determining positional information about an ana- 
tomical feature, as in Claim 13, where the anatomical feature is lo- 
cated in at lealst one of the images by computer imago'processing. 

15. A method of ctetermining positional information about an ana- 
tomical feature, as ih Claim 13, where the first Antage point is the 
position of the firsK surgical instrument, lacing one lens of a 
stereoscopic camera, a\d the second vantage point is the position 
of the second surgical instrument, being tffe position of the second 
lens of a stereoscopic camera. / 

16. A method of determining pteiti^nal information about an ana- 
tomical feature, as in Claim 13, Were 

the first and second surgical/fnstru\ents are the same, having a 
means for transmitting an image out V the patient's body, where 
the first image of the anatomical featurcN^ obtained by placing the 
surgical instrument at thfc first vantage pomt: 

and / \ 

the second image of the anatomical feature is obtained by moving 
the surgical instrument to the second vantage pointX 

17. A mcj/fiod of determining positional information about an ana- 
torn ical/eatu re, as in Claim 13, in which the first image L^obtaincd 
from aMlrst surgical instrument placed at the first vantage point and 
the Second image is obtained from a second surgical instrument at 
the second vantage point, both surgical instiumcnts having mcHjis 
m x transmitting images out of the patient's body. 

18. A method of controlling the position of a surgical instrument 
inside a patient's body the comprising steps of: 

inserting a first surgical instrument into the patient's body, the in- 
strument having a means for transmitting an image out of the pa- 
tient's body; 
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designating an anatomical feature of interest; 

transmitting an image of the designated anatomical feature out of 
the patient's body; 

determining precise 3D positional information about the designated 
anatomical feature of interest relative to the first surgical instru- 
ment; 

using the positional information to reposition the first surgical in- 
strument to a desired positional relationship relative to the ana- 
tomical feature. 

19. A method of controlling the position of a surgical instrument 
inside a patient's body, as in Claim 18, where the positional infor- 
mation about the feature of interest relative to the first surgical in- 
strument is obtained by means of image processing. 

20. A method of controlling the position of a surgical instrument 
inside a patient's body, as in Claim 18, where the positional infor- 
mation about the anatomical feature of interest relative to the first 
surgical instrument is obtained by manipulating a graphics object 
superimposed on an image of the anatomical feature. 

21. A method of controlling the position of a surgical instrument 
inside a patient's body, as in Claim 18, where (he positional infor- 
mation is used to reposition a second surgical instrument. 

22. A method of controlling the position of a surgical instrument 
inside a patient's body, as in Claim 18,-furthcr comprising steps of: 

designating a desired position of an anatomical feature relative to 
images transmitted out of the patient's body by the first surgical 
instrument; 

moving the first surgical instrument to a vantage point from which 
the designated anatomical feature is at the desired position in im- 
ages transmitted by the fiist surgical instrument, the first surgical 
instrument icmaining at a constant distance from the designated 
anatomical feature. 

23. A method of controlling the position of a surgical instrument 
inside a patient's body, as in Claim 18, further comprising steps of: 
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designating a desired direction of motion of an anatomical feature 
relative to images transmitted out of the patient's body by the first 
surgical instrument; 

moving the first surgical instrument so that it remains at a constant 
distance from the designated anatomical feature while causing the 
motion of the designated feature in images transmitted from the 
first surgical instrument to move in the desired direction. 

24. A method of controlling the position of a surgical instrument 
inside a patient's body, comprising steps of: 

inserting a first surgical instrument into the patient's body, the in- 
strument having a means for transmitting an image out of the pa- 
tient's body; 

designating an increment of motion along an axis of view of the first 
surgical instrument, the axis defined by a line from the vantage 
point of the surgical instrument lo the point on the patient's anat- 
omy that appears in the center of the image transmitted by the first 
surgical instrument; 

moving the first surgical instrument by the designated increment of 
motion along the axis of view, so that the point on the patient's 
anatomy appearing in the center of the image remains unchanged. 

* %5. A system for positioning a surgical instrumcnt-rclativc to a na 
tichj/s body, comprising: 

a robotictqanipulator having at least one controilcd^grcc of free- 
dom; X. 

a control means foiNunlrolling the manj^tJlator motions; 

instrument holding mcansN^r aKaching a first surgical instrument 
to the manipulator; >^x. 

a surgeon input mpsfns pctmitting the surgeon to specify desired 
motions of thc^rfgical instrument to thcXontrol means, where said 
input mcanxfuc mounted on a surgical instrument. 

26^a system for positioning a surgical instrument relative to a pa- 
rent's body, as in Claim 25, in which 
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the mbutie manipulator has a romoto center of motion distal to the ,, 
manipulator mechanism; ^^^^^"^ 

the instrument holding meansJi^W-^h^TTrst surgical instrument so 
that the point at which^ surgical instrument enters the pa- 
tient'sboii^rloc^ at the center-of-motion of the manipulator 
jftcCfTariism. 



27, A method of controlling the position of a surgical instrument 
inside a patient's body, comprising steps of: 

inserting a first surgical instrument into the patient's body, the in- 
strument having a means for transmitting an image out of the pa- 
tient's body; 

transmitting an image out of (he patient's body; 

displaying the transmitted image on a video screen; 

manipulating a computer graphics object superimposed on the video 
screen to designate a view of an anatomical feature; 

moving the first surgical instrument to a vantage point from which 
an image comprising the designated view may be obtained. 

A method of creating an image of an anatomical featu i c wkj^rfi 
a patient's body comprising the steps of: / 

inserting a first surgical instrument into the palicnt's^ody, the in- 
strument having a means for transmitting imag^out of the pa- 
tient's body; 

transmitting an image out of the paticn^Tbody; 

creating a wide-angle image of j^tanatomical feature within of the 
patient's body as it would *mp£ar from the vantage point of the fiist 
surgical instrument; 

replacing a portir^Tof the wide-angle image of the anatomical fea- 
ture with an ipr^gc transmitted by the first surgical instrument. 

29. A method of creating an image of an anatomical featuic within 
a natficnt's body, as in Claim 28, in which the wide-angle image is 
Xcomputcr-graphic rendering of the interior of the patient's body. 
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iO. A method of creating an image of an aualumical featu i e within 
aNpatient's body, as in Claim 29, in which the wide-angle image/is 
produced from at least one image transmitted from the interior of 
the patient's body. / 



31. A method of creating a stereoscopic image of an anatomical 
feature within a patient's body comprising the steps of: 



inserting a fVst surgical instrument into the patient's /body, the in- 
strument having a means for transmitting images />ut of the pa- 
tient's body; 



obtaining a first irf^gc of an anatomical feature/^ seen from a first 
vantage point; 

placing the first surgical instrument at a second vantage point at a 
known displacement relative to the first vantage point; 

transmitting a sequence of second images out of the patient's body, 
as seen from the second vantage poitft; 

presenting the first image as the View to one eye and the sequence 
of second images as the view lo/btW eye in a stereoscopic viewing 
system. 



32. A method of creating 
feature within a paticntVbody, as in Cf 
image is a computer-graphic rendering o 
ticnt's body. / 



stereoscopic image of an anatomical 
im 31, in which the first 
the interior of the pa- 



33. A method of cheating a stereoscopic imag\of an anatomical 
features within a t/aticnt's body, as in Claim 31, ii\which the images 
displayed compile computer-graphic renderings of\cdgcs extracted 
from the imagps transmitted from the infciior of thcValicnt's body. 



34. A syslpfn for positioning a surgical instrument relative to a pa- 
tient's bod^, as in Claim 25, in which the said surgeon inftut means 
comprisc/a joystick. 

35. A^systcm for positioning a surgical instrument relative to & pa- 
tients body, as in Claim 25, in which the said surgeon input mcins 
comprise a force sensing clement mounted on a surgical instrument 
hfeld by the surgeon. 
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Vi - A gycaf^m fnr pnrifinninn n rnrpirol inrtnimunt r^loth^tr^ po_ 

tient's body, as in Claim 25, in which thesaid_^iicgemr1nputm 
are mounted on a surgicajJnsJxiHftCTTT^hTch to be inserted into 
the patiejit^Jiodyr^^^ means being mounted on the 

peilftm^oTthc instrument which remains outside the patient's bod}'. 
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ABSTRACT 



The present method and apparatus use image processing to deter- 
mine information about the position of a designated object. The 
invention is particularly useful in applications where the object is 
difficult to view or locate. In particular, the invention is used in 
endoscopic surgery to determine positional information about an 
anatomical feature within a patient's body. The positional infor- 
mation is then used to position or reposition an instrument (surgical 
instrument) in relation to the designated object (anatomical fea- 
ture). 

The invention comprises an instrument which is placed in relation 
to the designated object and which is capable of sending informa- 
tion about the object to a computer. Image processing methods arc 
used to generated images of the object and determine positional in- 
formation about it. This information can be used as input to 
robotic devices or can be rendered, in various ways (video graphics, 
speech synthesis), to a human user. Various input apparatus arc 
attached to the transmitting or other used instruments to provide 
control inputs to the computer. 
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